Method and apparatus for preventing overloads of power distribution networks

ABSTRACT

Systems and methods for monitoring power in power distribution systems are provided. In one aspect, a system for monitoring power includes a power monitoring device that measures a value of at least one characteristic of power provided to a branch of a power distribution system. The power monitoring device includes an output that provides the value measured. The system further includes a controller having an input to receive the value measured and an output that couples to a first device powered by the branch to send a maximum power signal to the first device to command the first device to operate at a percentage of maximum power.

FIELD OF THE INVENTION

The present invention relates generally to methods and apparatus formeasuring current or power delivered to loads in power distributionnetworks, and more specifically, to methods and apparatus for preventingoverload conditions in power distribution equipment used to powerequipment having variable input power requirements.

BACKGROUND OF THE INVENTION

The proliferation of the Internet has created a need for large scaledata centers that contain tens, if not hundreds, of racks of computingequipment, such as servers and routers. One of the major problemsconfronted by designers of these data centers is the requirement toroute facility power to each of these racks of equipment. Typically,branch circuits from a primary or a secondary distribution panel arerouted to groups of racks to provide power to the equipment in the rack.Each of the branch circuits is designed to provide a predeterminedmaximum power level or current, and the size of cabling used to routethe power for a branch, and the size of circuit breakers used for thebranch are selected based on this predetermined maximum power level or,more typically, predetermined maximum current.

Typically, it is desirable to design each of the branch circuits suchthat the total current drawn by the equipment coupled to any given oneof the branch circuits is at some predetermined percentage (for example50%) of the maximum current level for that branch circuit. This allowssome flexibility to add additional equipment to racks and provides asafety margin below the maximum current level.

To properly design the routing of the branch circuits, it is desirableto know, with some accuracy, the current that is drawn by the equipmentin the racks. Traditionally, the power or current drawn by computerequipment could be determined based on manufacturers' specificationsand/or by making actual measurements of the current being drawn by theequipment.

These measurements and specifications are only useful for equipment forwhich the current draw is substantially static, which in the past wastrue for typical computing equipment. However, for newer computingequipment, the current draw is typically not static due to a number offactors including: 1) many computers utilize some form of powermanagement strategy which minimizes the power (and current) consumptionof the computer by turning off or slowing down subsystems within thecomputer when they are not in use; 2) cooling systems (i.e., fans) areoften speed controlled based on air and component temperatures to reducepower consumption and acoustic noise generation; and 3) the amount ofpower drawn by the processors and memory systems in computers hasincreased steadily with the increase of speed of the processors, so thatthe power consumed by the processors and memory subsystems may exceed50% of the total power draw of a computer. The power drawn by processorsand memory systems is variable depending on the processing load, andsince the total power of these systems may be a significant portion ofthe total power, the total power draw of a computer can varysignificantly depending on the processing load on the computer.

The operating systems of most computers are capable of simultaneouslyperforming multiple tasks by assigning segments of the CPU processingtime to each of the tasks on a priority basis. Any remaining segments ofthe CPU processing time are occupied by an idle task in which the CPUcan be halted and all associated clocks can be stopped to reduce thepower draw of the computer. Further, some computers, for example, thosethat utilize the Windows® 98 or Windows® 2000 operating system, have anAdvanced Control and Power Interface (ACPI) feature that allows theoperating system to control power to fans and other devices in thecomputer to further reduce the power drawn by the computer. Because ofthe factors described above, it is not unusual for a more modern systemto consume twice as much power when the processors are fullycomputationally loaded and operating in a warm environment, then whencomputationally idle and operating in a cool environment.

The variability of the power draw of computers complicates theelectrical design of data centers. Computer manufacturers typicallyprovide power ratings on nameplates. These nameplate values aretypically maximum values that are determined based on the maximum powerthat a computer may draw when fully loaded with all options and with allsubsystems operating at full load. Because of conservative approachestaken in determining nameplate values, they are often greater than evenworst case values for a given computer, and accordingly are of littleuse to an electrical facility designer. While a designer may measure thecurrent drawn by a computer or a set of computers to determine the powerdraw, it is typically not known at the measurement time, whether thecomputer is at full load or at what percentage of full load the computeris operating.

Several problems may occur when circuit branches are designed based onmeasured power draw values of computers. First, the wiring used in powerrouting circuits may be undersized for full load conditions, and second,when one or more of the computers powered from a branch are operated atfull load, the current drawn may exceed the circuit breaker value forthe branch, causing the circuit breaker to trip and disconnect power tothe computers. For critical applications of computers, any such powerinterruption is often unacceptable. Further, to prevent powerinterruptions to critical computers, it is common to use uninterruptiblepower supplies (UPSs) for these computers. Often, one UPS is used topower multiple computers or racks of computers. To properly size theUPS, it is necessary to know the power draw of each of the computers andother equipment powered by the UPS. The variability of the power draw innewer computers makes it difficult to properly size a UPS for theseapplications.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide improved systems andmethods for measuring the current or power draw of computers and racksof equipment that overcome problems described above.

A first aspect of the present invention is directed to a system formonitoring power in a power distribution system. The system includes apower monitoring device located in the power distribution system tomeasure a value of at least one characteristic of power provided to abranch of the power distribution system, the power monitoring devicehaving an output that provides the value measured, and a controllerhaving an input to receive the value measured and an output that couplesto a first device powered by the branch of the power distribution systemto send a power signal to the first device to command the first deviceto operate at a predetermined percentage of maximum power.

The system for monitoring power can further include a plurality of powermonitoring devices, each located in the power distribution system tomeasure at least one characteristic of power provided to a respectivebranch of the power distribution system, and each having an output tocouple to the controller to provide a value of the characteristicmeasured. Each of the respective branches of the power distributionsystem can provide power to at least one respective device, and thecontroller can be adapted to send a power signal to each respectivedevice to command each device to operate at the predetermined percentageof maximum power. The controller can be adapted to send the power signalto devices powered by one branch at a same time, to cause each of thedevices on the one branch to operate at the predetermined percentage ofmaximum power. The controller can be adapted to total the valuesmeasured for each of a plurality of branch circuits and compare thetotal with a first overload value to detect an overload condition. Thecontroller can be adapted to send an alarm signal to an operator upondetection of an overload condition. The controller can be adapted tosend a signal to disconnect power to one or more devices upon detectionof an overload condition. The at least one characteristic can beelectrical current.

The controller of the power monitoring system can further include afirst network interface to communicate with devices powered by the powerdistribution system over a first communications network and a secondnetwork interface to communicate over a second communications network.Each of the plurality of power monitoring devices can include a networkinterface to communicate with the controller over the secondcommunications network. The power distribution system can include anuninterruptible power supply, and the controller can be adapted tocommunicate with the uninterruptible power supply to detect that theuninterruptible power supply is operating on battery mode and replacethe first overload value with a second overload value. The controllercan be adapted to send a signal to interrupt power to at least onedevice upon detection that the uninterruptible power supply is operatingon battery mode. The system can further include a plurality oftemperature sensors that monitor temperature at locations within afacility, each of the temperature sensors having an output tocommunicate a temperature value to the controller. The controller can beadapted to compare temperature values received from the temperaturesensors with predetermined values to detect an over temperature errorcondition, and upon detection of an over temperature error conditionsend an alarm signal. The controller can be adapted to send a signal tointerrupt power to at least one device upon detection of an overtemperature error condition. The predetermined percentage of maximumpower can be 100 percent.

Another aspect of the present invention is directed to a method formonitoring and controlling a power distribution system that has aplurality of circuit branches for providing power to a plurality ofdevices. The method includes controlling a first device on a firstcircuit branch to operate at a predetermined percentage of maximumpower, detecting a first value for a characteristic of power provided tothe first circuit branch, controlling a second device on a secondcircuit branch to operate at a predetermined percentage of maximumpower, detecting a second value for a characteristic of power providedto the second circuit branch, adding the first value to the second valueto obtain a total value, comparing the total value to an overload valueto detect an overload condition, and indicating an alarm condition whenthe total value exceeds the overload value.

The first device can be controlled to operate at less than thepredetermined percentage of maximum power when the second device iscontrolled to operate at the predetermined percentage of maximum power.The method can further include controlling one of the plurality ofdevices to operate in a reduced power mode upon detection of an overloadcondition. The method can further include interrupting power to one ofthe plurality of devices upon detection of an overload condition. Thecharacteristic measured can be electrical current. The method canfurther include communicating with the first device and the seconddevice over a first communications network, and communicating with powerdetection devices over a second communications network. The powerdistribution system can further include an uninterruptible power supply,and the method can further include detecting when the uninterruptiblepower supply is operating in a battery mode, and controlling at leastone of the plurality of devices to operate in a reduced power mode upondetection of the battery mode. The method can further includeinterrupting power to at least one of the plurality of devices upondetection of the battery mode. The power distribution system can be atleast partially contained within a facility, and the method can furtherinclude measuring air temperature at a plurality of locations within thefacility, comparing at least one value of air temperature measured witha predetermined value to detect an over temperature condition, andcontrolling at least one of the plurality of devices to operate in areduced power mode upon detection of the over temperature condition. Thepredetermined percentage of maximum power can be 100 percent.

Yet another aspect of the present invention is directed to a system formonitoring and controlling a power distribution system that has aplurality of circuit branches for providing power to a plurality ofdevices. The system includes means for controlling each of the pluralityof devices to operate at a predetermined percentage of maximum power,and means for detecting a value of a characteristic of power provided toeach of the plurality of circuit branches.

The system can further include means for comparing a total value of thecharacteristic with a predefined value to detect an overload condition.The system can further include means for interrupting power to at leastone of the plurality of devices when an overload condition is detected.The characteristic can be electrical current. The power distributionsystem can include at least one uninterruptible power supply, and thesystem can further include means for detecting that the uninterruptiblepower supply is in a battery mode of operation, and means for adjustingthe predefined value when the uninterruptible power supply is in thebattery mode of operation. The system can further include means fordetecting air temperature values in a facility containing the powerdistribution system. The system can further include means for comparingthe detected air temperature values with predetermined temperaturevalues, and means for interrupting power to at least one of theplurality of devices when the detected air temperature values exceed thepredetermined temperature values. The predetermined percentage of powercan be 100 percent.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention, reference is madeto the drawings which are incorporated herein by reference and in which:

FIG. 1 shows a typical layout of the power distribution system in a datacenter;

FIG. 2 shows a power distribution control system in accordance with afirst embodiment of the present invention;

FIG. 3 shows the power distribution control system of FIG. 2 operativelycoupled to the power distribution system of FIG. 1.

FIG. 4 shows a flow chart of a method of controlling power flow in apower distribution system in accordance with one embodiment of thepresent invention;

FIG. 5 shows a power distribution control system of a second embodimentof the present invention; and

FIG. 6 shows a power distribution control system of a third embodimentof the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention that provide methods and systemsfor monitoring and controlling power distribution in data centers willnow be described. As understood by those skilled in the art, embodimentsof the present invention are not limited for use in data centers, butmay also be used in other facilities in which it is desired to monitorand control power distribution. Further, embodiments of the presentinvention may also be used aboard ships, airplanes or other mobileplatforms where it is desired to monitor and control power distribution.

FIG. 1 provides a diagram of a typical power distribution system 100 fora data center. The power distribution system 100 includes a facilityinput power port 102, an uninterruptible power supply (UPS) 104, a powerdistribution unit (PDU) 106, a 108, and three power distributionbranches 110, 112 and 114. In FIG. 1, the power distribution system isused to power nine loads 116 a-116 i. As is known to those skilled inthe art, some of the components shown for the power distribution systemare optional. For example, the UPS is an optional component that is usedto provide power to the loads in the event of disruption of the facilitypower.

The PDU 106 may include a number of power devices such as switches, atransformer and may include circuit breakers in addition to or in placeof circuit breakers contained in the circuit breaker panel 108. Thecircuit breaker panel 108 distributes power from the PDU to each of thethree power distribution branches 110, 112 and 114, and provides circuitbreaker protection for each of the power distribution branches. Thepower loads 116 a-16 i may be equipment racks containing multiplecomputers, standalone computers, standalone mass storage devices, or anyother equipment that is typically found in a data center.

Embodiments of the present invention provide power monitoring andcontrol for power distribution systems such as power distribution system100. FIG. 2 provides a block diagram of a power monitoring and controlsystem 200 of the present invention. The system includes a centralcontroller 201, a first set of power monitoring devices 202A, 202B, and202C, a second set of power monitoring devices 204A, 204B and 204C, anintelligent power strip 205, a consolidator 206, a first computer 208, asecond computer 210, and a network 211 operatively coupling thecomponents of the system 200.

With the exception of containing a power control module 212, the firstcomputer and the second computer are standard computers that aretypically found in a data center and may be functioning as servers,routers or in some other capacity, and may be mounted in racks of thedata center. In different embodiments of the present invention, thepower control module 212 is implemented using software, hardware, or acombination of software and hardware. In one embodiment the powercontrol module is configured to respond to a signal received over thenetwork 211 and control the computer in which it resides to consumemaximum power by ensuring that all of the subsystems within the computerare operating at substantially 100%, all cooling systems are operatingat 100%, and the computational load on all processors is at 100%. Inother embodiments, the power can be controlled at a predeterminedpercentage of maximum power.

In some embodiments of the present invention, the power control moduleis designed to increase the power draw of a computer to maximum, withoutsubstantially interfering with tasks being performed by the computer. Inparticular, in one embodiment of the present invention, that will now bedescribed, the power control module is configured to operate withcomputers that conserve power by utilizing an idle task (described inthe background section above). For this embodiment, the power controlmodule creates an additional task that is assigned a priority levellower than all other tasks being processed by the CPU, but higher thanthe idle task. The additional task is designed to fully utilize the CPUand all disk drives, memories and other devices within the computer forthe entire idle time of the CPU to maximize power consumption by thecomputer. To ensure that the additional task has the appropriatepriority, the control module may reconfigure the priorities of each ofthe other tasks. For computers that have multiple processing units, eachof the processing units is configured to operate at maximum capacity. Byoperating as described above, the control module is able to cause acomputer to operate at full, or near full power, without affecting theoperation of tasks being performed by the computer. Further, in oneparticular embodiment, for use with computers that utilize ACPI or othersimilar means to control power draw of the computers, the power controlmodule may be configured to control the speed of fans and the operationof other devices to operate at maximum power when desired.

Each of the power monitoring devices of the first set 202A, 202B and202C and each of the power monitoring devices of the second set 204A,204B, 204C are inline power monitoring devices that, in one embodiment,measure the current of a particular power feed. In other embodiments,the power monitoring devices may directly measure power. The powermonitoring devices of the first set are intelligent devices that haveinternal network interface circuitry to enable the devices tocommunicate with other devices, such as the controller 201, over thenetwork 211. The power monitoring devices of the second set have lessintelligence than the devices of the first set and do not have theability to communicate directly with the network 211. In one embodiment,the monitoring devices of the second set utilize point to pointsignaling such as RS-232 to communicate power levels to the consolidator206. In other embodiments, the monitoring devices of the second set mayutilize a network scheme or bus scheme like an RS-485 multi-drop bus, apower line carrier network, a Controller Area Network (CAN) bus, or aLONWORKS® twisted pair network, to communicate with the consolidator.

The consolidator 206 has multiple logical inputs to receive the signalsfrom each of the power monitoring devices of the second set. Inaddition, the consolidator has a network interface to allow it tocommunicate with the network 211. In embodiments of the presentinvention, the consolidator receives data indicative of power levelsmeasured by each of the power monitoring devices coupled to it andforwards these levels to the controller 201 over the network, along withidentifying information for each of the devices. In one embodiment, theconsolidator is a rack mounted device that may be mounted in one of thecomputer racks in a data center. In another embodiment, the consolidatormay be implemented using a computer that also performs other functionsin a data center or other facility.

The intelligent power strip 205 is a power strip having multiple poweroutlets and current monitoring devices incorporated within it fordetermining the current draw of any one of the power outlets or thetotal current draw of all devices that are powered from the intelligentpower strip. The intelligent power strip also includes network interfacecircuitry to allow the strip to communicate with the controller 201 overthe network 211. In addition, the intelligent power strip can becommanded by the controller to interrupt power to any of the poweroutlets on the strip. In one embodiment, the intelligent power strip maybe implemented using a Masterswitch VM® power strip, available fromAmerican Power Corporation of West Kingston R.I., that has thecapability of measuring total current drawn by devices powered throughthe device and the capability to control the application of power toindividual outlets.

In embodiments of the present invention, the controller 201 functions asthe central controller for the system and communicates with othercomponents of the system over the network 211. In one embodiment, thecontroller 201 includes a power load monitoring and control module 216that communicates with the other components of the system to control theother components and receive power draw levels or current draw valuesfrom the power monitoring devices. The controller may be implementedusing a single computer contained in one of the racks of a data center,using a desktop computer, a dedicated purpose computing device, anembedded computing system, or the functionality of the controller may bedistributed among several networked computers. The control module 216may be implemented using software, hardware or a combination of softwareand hardware.

The network 211 provides the connectivity between the components of thesystem. In one embodiment, the network may be implemented using one of anumber of well known network architectures such as an Ethernet network.The network 211 may also be used by the first computer 208 and thesecond computer 210 to communicate with other devices within a datacenter or to communicate with devices outside of the data center over,for example, the Internet.

The system 200 of FIG. 2 may be implemented in the power distributionsystem of a data center as will now be described with reference to FIG.3, which shows the system 200 of FIG. 2 implemented in the powerdistribution system 100 of FIG. 1. As shown in FIG. 3, a powermonitoring device (202A-202M of 204A-204C) is incorporated at a numberof places in the power distribution system 100. Two additional loads 117and 119 are included in the system of FIG. 3. Load 117 is an equipmentrack that contains the controller 201 and the consolidator 206, however,as understood by those skilled in the art, the controller andconsolidator could be located in separate racks or need not be installedin a rack at all. Load 119 is also an equipment rack containing thefirst computer 208, the second computer 210 and the intelligent powerstrip 205. The diagram of FIG. 3 shows only the power connectionsbetween the components. The signal connections between the components ofthe power monitoring system are as shown in FIG. 2. In the embodimentshown, the controller 201 is powered from the power distribution systemthat the controller is monitoring and controlling. In other embodiments,the controller may be powered by a separate power distribution system.

In the embodiment of the present invention shown in FIG. 3, a powerdistribution device is placed to measure the current drawn by eachcomputer or server, by each rack, and on each branch circuit at theoutput of the circuit breaker panel. Additional power monitoring devicescould be added at other points in the power distribution system, or ifless monitoring is desired, fewer power monitoring devices could beused. In one embodiment, each of the power monitoring devices may bedefined as having an order value with respect to a given point in thepower distribution system. The order value for a given power monitoringdevice is determined based on the number of power monitoring devicesthat are in the power distribution system between the given device andthe given point. For example, with reference to FIG. 3, for circuitbranch 112 and with the circuit breaker panel 108 as the referencepoint, power, monitoring device 202C has an order of one, and powermonitoring devices 202H, 204A, 204B and 204C have an order of two, andeach of the power monitoring devices in the intelligent power strip 205has an order of three.

Methods of controlling and monitoring the power and/or current in powerdistribution systems using the systems described above will now bedescribed with reference to FIG. 4. However, methods of the presentinvention are not limited for use with the above-described systems, butmay be used with other systems as well.

In one embodiment of a method 300 of the present invention, which issummarized in flowchart form in FIG. 4, the method determines themaximum current draw for each of the circuit branches of a powerdistribution system. The maximum current draw can be compared topredetermined values that are based, for example on circuit breakervalues, and if a potential overload condition is detected, warnings canbe generated and corrective actions can be taken. In a first step 302 ofthe method 300, the layout of the power distribution system is enteredinto the controller and a reference point is chosen. In one embodiment,the power monitoring and control module includes a program for enablinga user to enter the layout through a graphical user interface (GUI). Inother embodiments, the power monitoring and control module is capable ofreceiving a data file containing the layout created using one of anumber of computer aided design programs such as, for example, Visio®,Autocad® or a custom designed program.

Once the layout has been entered and the reference point has beenchosen, in step 304, one of the circuit branches of the powerdistribution system is selected for analysis. Next, in step 306, anorder value is set equal to the highest order value of all powermonitoring devices in the selected circuit branch. For example, if thechosen circuit branch is branch 110 (FIG. 3), then the order value isset to that of devices 202J, 202K, 202L and 202M. In step 308, one ofthe power monitoring devices having the set order value is chosen. Instep 310 all equipment that is powered through the selected powermonitoring device is controlled to draw maximum power either manually,or automatically over the network 212 by the controller 201. Once all ofthe equipment is drawing maximum power, the selected power monitoringdevice communicates a value of power draw or current measured to thecontroller 201 in step 312. Next, in step 314, the equipment coupled tothe selected power monitoring device is returned to its prior state.

In step 316 of method 300, a determination is made as to whether thereare any other power monitoring devices in the selected circuit branch ofthe selected order that have not yet been selected. If the outcome ofstep 316 is YES, then another power monitoring device of the same orderis selected, and steps 308 to 314 are repeated. For circuit branch 110,there are a total of four monitors having the highest order for thebranch, and therefore, steps 308 to 314 will be repeated four timesuntil the outcome of step 316 is “NO”.

If the outcome of step 316 is NO, then in step 318 a determination ismade as to whether there are any devices having an order value less thanthe selected device. If the outcome of step 318 is YES, then the setorder value is reduced by 1 in step 319, a device having the nexthighest order value is selected, and steps 308 to 316 are repeated. Ifthe outcome of step 318 is NO, then a determination is made at step 320as to whether all branches have been measured. If all branches have beenmeasured, then the process ends at 322. If all branches have not beenmeasured, then steps 304 to 318 are repeated for another branch.

Once all of the maximum current draw values have been determined, thetotal maximum current draw can be compared to predetermined values todetermine whether any corrective action should be taken. Additionally,the maximum current draw at each component or element (i.e., a circuitbreaker, fuse or other device) having a maximum current rating can becompared to the current rating to determine if it is necessary to takeany corrective actions. Corrective actions may include adding a branchcircuit, moving equipment from one branch circuit to another branchcircuit, or one of a number of other actions. In a system in which a UPSis used, the corrective action may include adding an additional UPS to abranch or adding additional capacity to an existing UPS.

In the method described above, the power draw on circuit branches ismeasured successively. In another embodiment, power draw may be measuredon multiple circuit branches simultaneously. In this embodiment, theduration of maximum power for devices may be kept at a minimum to reducethe likelihood of tripping a circuit breaker serving two or more circuitbranches that are measured simultaneously. As is well known, a typicalcircuit breaker will not trip instantly when the current exceeds thebreaker's threshold, but typically will only trip when the excesscurrent is maintained for some period of time.

In method 300 described above, the power draw is successively measuredat power monitoring devices of lower order. In one embodiment of thepresent invention, prior to maximizing the power draw of all equipmentpowered through a particular branch circuit having a power monitoringdevice, the total power draw determined using all higher order powermonitoring devices in the same branch is determined. The total is thencompared with known allowable maximum levels for all sower order devicesto ensure that the simultaneous powering of all higher order equipmentat maximum levels will not cause power draw levels in excess of safe,allowed maximum values. If it is determined that the simultaneouspowering may cause levels to be above allowed maximum values, then oneof the corrective actions described below may be taken.

In embodiments of the present invention, one power monitor may bepositioned to measure the power drawn by a plurality of devices withouta higher order power monitor installed between the power monitor and anyof the devices. In such a situation it may be undesirable tosimultaneously bring all the devices to maximum power to measure themaximum power draw. In one embodiment, the devices are operated atmaximum power one at a time with the other devices powered off to obtainthe power draw for each device, and then the individual power draws aretotaled to obtain the maximum at the power monitor. However, in someinstances one or more of the devices may be running a criticalapplication that is intended to be run 24 hours per day, seven days perweek, without interruption. In such a situation, in one embodiment ofthe present invention, the maximum power draw for the combined devicesis determined as follows, using an example of a situation where onepower monitor is coupled to measure the power draw of three devices.

First, an ambient power measurement is made of the power or currentdrawn by the combination of the three devices as presently configuredand operating. Next, a first one of the three devices is individuallycontrolled to operate at maximum power while the other two (the secondand third devices) continue to operate at their present state, andanother power or current measurement is made. Then, the first device isreturned to its normal operating state and another ambient powermeasurement is made of the combined power or current draw from thecombination. If the ambient power measurement is substantially the samebefore and after the first device was configured to operate at maximumpower, then an assumption is made that the power or current draw of thesecond and third devices remained substantially constant during the timethat the first device was configured to draw full power. The increase inpower or current draw over the ambient value contributed by configuringthe first device to operate at maximum power is then determined bysubtracting either ambient value from the value measured with the firstdevice operating at full power.

The increase power over the ambient value for maximum power draw for thesecond and third devices can be determined in the same manner as thefirst device described above. Then the total maximum power can bedetermined by adding the increase for each of the three devices to theambient value. In situations where the ambient value does not staysubstantially constant before and after increasing the power draw of oneof the other devices, then in one embodiment of the present invention,the procedure is repeated a number of times, and if the ambient valuestill does not stay constant, then the ambient value which produces thegreatest increase is used. The increase for each of the three can thenbe added to the worst case ambient value. If the resulting value iswithin acceptable limits, all three devices may then be controlled tooperate at maximum power draw, and an actual measurement with all threedevices at maximum power can be made. If it is determined that thesimultaneous powering may cause levels to above allowed maximum values,then one of the corrective actions described below may be taken.

The method 300 described above may be performed when equipment is firstinstalled, when additional equipment is to be added to a system, or themethod may be performed periodically as part of a scheduled maintenanceprogram. In another embodiment of the present invention, the controllerprovides for constant monitoring of the power draw or current at each ofthe power monitoring devices to detect an actual or potential overloadcondition. Present values of power draw can be compared to predeterminedlimits that are calculated based on previously conducted measurements,circuit limitations, or other factors. Rather than performing constantmonitoring, embodiments of the present invention also provide forperiodic measurements or scheduled measurements.

When a potential overload condition is detected, one of a number ofactions, or a combination of actions, may be initiated by the controller201. These actions include sending notifications and logging problems aswell as taking corrective actions. The notifications can includerecording an event in a log and activating an audio or visual alarm.Further, the notifications may include sending an email to a systemadministrator or facility manager or paging the administrator ormanager. Still further, in some embodiments, the controller may send asignal, such as an SNMP trap, to another computer to notify the othercomputer or its operator of the condition.

In addition, when a potential overload condition is detected, thecontroller may take positive steps to ensure that an overload will notoccur. In one embodiment, the controller may initiate a shutdown commandof one or more computers by communicating with the computers over thenetwork, or the controller may command one or more computers to operatein a mode that draws less power. In other embodiments, the controllermay also communicate with an intelligent power strip to command thepower strip to interrupt power to one or more of its outlets. In stillother embodiments, all or some of the power monitoring devices include apower interruption mechanism that can be activated by the controllerover the network to interrupt equipment powered through the powermonitoring device. By selectively powering off lower priority devices,the controller can ensure that power continues to be provided tocomputers that are running higher priority applications. Further, newdevices may be prevented from being powered on by automaticallyswitching off the power to outlet strips or individual outlets of outletstrips. After taking one of the above actions to prevent an overload,the controller can determine whether an overload potential has beenavoided, and if not, can take further steps to reduce the power draw.

In one embodiment, to minimize power draw to avoid an overload, thecontroller can control a computer device to operate at less than maximumload by instructing the computer to exercise a low power task thatutilizes CPU process time, but during that time, halts operation of theCPU. The low power task can be assigned a priority level higher thanother tasks on the computer to ensure that sufficient low power timeoccurs. The average power load of the computer can be maintained at somefixed percentage using this method.

In one embodiment, the controller is coupled to the UPS 104 over thenetwork to detect when the UPS has switched to battery mode. In responseto detecting that the UPS is operating on battery, the controllermeasures the power draw at the power monitoring devices, and may takeactions as described above to reduce the power draw to minimize thedrain on the UPS to provide power longer for critical applications.

In some data centers, it is known to provide a dual power feed or someother multiple power feed to equipment. The multiple power feedstypically provide redundancy and/or accommodate relatively high powerequipment that has multiple power feeds. To prevent overload whenmultiple feed power distribution systems are used, in one embodiment ofthe present invention, the measured maximum current of one feed isadjusted to account for an increase in current that will occur if one ofthe other feeds of the dual feed system fails. For example, in a dualfeed system for a device, in which each of the feeds equally shares thecurrent draw of the device, when determining if a potential overloadcondition exists, the controller multiplies the maximum current measuredon one of the feeds by two to estimate the load on the feed upon failureof the other feed.

In another embodiment of the present invention, a system 400 monitorsair temperature at various locations as well as provides the functionsof the power distribution monitoring and control system 201. The system400, as shown in FIG. 5, includes all of the components of system 200plus additional sensors 402A and 402B for measuring air temperature in afacility. Sensors 402A and 402B are coupled to the controller to allowthe controller to detect hot spots and take corrective action. Furthertwo intelligent air conditioning systems 404 and 406 are also coupled tothe controller 201 over the network 212, and may also be coupleddirectly to sensors 402A and 402B. The controller, in response todetecting potential or actual cooling problems can control the airconditioning systems to increase their outputs or redirect their outputsto prevent problems. In one embodiment, the loads are controlled tooperate at maximum power for an extended period of time, while the airtemperature is being monitored, to ensure that the air conditioningsystem is capable of supplying sufficient cool air for the maximumrequirements. In other embodiments, more or less temperature controllersand air conditioning systems may be incorporated into the system 400.

In one embodiment, the system 400 may further include an additionalsensor, identified as sensor 402C in FIG. 5, located outside of thefacility to detect the outside temperature. As understood by thoseskilled in the art, the efficiency of many air conditioning systems isdependent on outside air temperature. When determining whethersufficient cooling is available from the air conditioning units, thecontroller can account for changes in efficiency of the air conditioningunits caused by changes in the outside air temperature.

In another embodiment of the present invention, which will now bedescribed with reference to FIG. 6, a power monitoring and controlsystem 500 is provided. System 500 is similar to system 400, except thata second network 213 is provided. The second network 213 is used toprovide communications between the power monitors of the system, thetemperature sensors of the system and the controller 201. As shown inFIG. 6, the controller 201, each of the power monitoring devices of thefirst set 202A, 202B, and 202C, the consolidator 206, the intelligentpower strip 205, temperature sensors 402A, 402B and 402C and airconditioning units 404 and 406 are all interconnected by the secondnetwork 213. Also, as indicated by the dotted lines in FIG. 6, inaddition to being coupled to network 211, each of computers 208 and 210may optionally be coupled to the second network 213 in addition to thefirst network 211. In different embodiments of the present invention,computers 208 and 210 can communicate with the controller 201 over thefirst network or the second network or over both the first network andthe second network.

The second network may be implemented using one of a number of networktypes, such as an Ethernet network or a power line carrier network. Inone embodiment, the second network is a private network that uses amodified version of the EIA-721 Common Application Standard (CAL) overIP in addition to SNMP and HTTP. The use of the second network providesseveral advantages. First, in embodiments of the present invention, thenumber of devices coupled to the second network is relatively low, andthe amount of data to be transmitted over the network is anticipated tobe relatively low. Accordingly, the software and hardware required ineach of the devices to communicate over the second network is not overlycomplex or expensive to implement. Second, the traffic on the secondnetwork is kept separate from the traffic on the first network, andtherefore, the traffic on the second network will not utilize bandwidthon the first network. In addition, the traffic on the second network issecure from users of the first network. This security becomesparticularly important for applications in which the first network iscoupled to the Internet and/or critical applications are operating inthe computer devices of the network. Another advantage to the use of thesecond network is that address space on the first network is notoccupied by the devices coupled to the second network.

In above-described embodiments, external power monitoring devices areused to monitor the power to computers or groups of computers. Inanother embodiment, some or all of the computers may have powermonitoring devices contained within, allowing the computers to monitortheir own power, and directly report their power draw to the controllerover the network 211.

Embodiments of the present invention described above are for use with ACpower distribution systems. However, the present invention is notlimited for use with AC power distribution systems but also may be usedwith DC power distribution systems. In addition, embodiments of thepresent invention may be used in data centers that utilize both AC andDC power distribution systems. As understood by those skilled in theart, when used with a DC system, several components of the ACembodiments described above may not be needed, such as a powerdistribution unit containing a transformer.

Embodiments of the present invention described above, overcome problemsassociated with designing and maintaining power distribution systems indata centers by providing more accurate monitoring and controllingcapabilities of the power draw of computer systems coupled to a powerdistribution system. In embodiments of the present invention describedabove, computer systems are controlled to operate at 100% of theirmaximum power to calculate the maximum power draw on circuit branches.As understood by those skilled in the art, in other embodiments,computers could be controlled to operate at known percentages of fullload (i.e., 50% of full load and 75% of full load) and scaling factorscould be used to extrapolate full load values based on measurements atknown operating points. Such a system is advantageous in that it may besafer to first operate a device at a known value less than full power todetermine if any problems may occur at full power before operating thedevice at full power.

In embodiments of the present invention described herein, currentmonitors are used to measure the current drawn by a given device or agroup of devices to determine whether maximum current or power valuesmay be exceeded in a system. As understood by those skilled in the art,the power drawn by a device is related to the current drawn by thatdevice, and embodiments of the present invention are not limited tosystems that utilize current monitors, but rather, also include systemsthat utilize monitors based on power and/or other electricalcharacteristics.

In embodiments of the present invention discussed above, an additionaltask having low priority is added to a task list of a computer to causethe computer to operate at maximum power. In other embodiments of thepresent invention, a task having a high priority may be added to cause acomputer to operate at a predetermined percentage of maximum power. Thetask that is added may cause the processor to be idle or to operate atmaximum capacity depending on whether it is desired to operate at a lowor high percentage of maximum power. For example, in one embodiment, acomputer can be controlled to operate at a minimum (or ambient) level bycausing the processor to be idle for nearly 100% of the processor timeby using a task that has a high priority, requires maximum processortime, and places the processor in an idle state. By causing a computerto operate at the ambient level and then the maximum level, the powerconsumption dynamic range of a computer can be determined. This dynamicrange may be used by power distribution system designers in designingfacilities.

Embodiments of the present invention are described above as beingimplemented with rack mounted computers. As known by those skilled inthe art, in some data centers, computer servers are implemented assingle cards, identified as server blades, installed within a commoncard cage or chassis, which is in turn typically installed in a rack.Embodiments of the present invention may also be used with server bladesto individually control the power draw of each server blade and tocontrol the combined power draw of two or more server blades installedin a common chassis.

In some embodiments of the present invention, as described above, thecontrol modules 212 in the computers 208 and 210 are described as beingimplemented by software or a combination of hardware and software. Inone embodiment, the control module is implemented as software that ispackaged and installed with UPS management software.

Having thus described at least one illustrative embodiment of theinvention, various alterations, modifications and improvements willreadily occur to those skilled in the art. Such alterations,modifications and improvements are intended to be within the scope andspirit of the invention. Accordingly, the foregoing description is byway of example only and is not intended as limiting. The invention'slimit is defined only in the following claims and the equivalentsthereto.

What is claimed is:
 1. A system for monitoring power in a powerdistribution system, the system comprising: a plurality of powermonitoring devices, each located in the power distribution system tomeasure at least one characteristic of power provided to a respectivebranch of the power distribution system, and each having an output toprovide a value of the characteristic measured; and a controller havingan input to receive the value measured and an output that couples to afirst device powered by the branch of the power distribution system tosend a power signal to the first device to command the first device tooperate at a predetermined percentage of maximum power.
 2. The system ofclaim 1, wherein each of the respective branches of the powerdistribution system provides power to at least one respective device,and wherein the controller is adapted to send a power signal to eachrespective device to command each device to operate at the predeterminedpercentage of maximum power.
 3. The system of claim 2, wherein thecontroller is adapted to send the power signal to devices powered by onebranch at a same time, to cause each of the devices on the one branch tooperate at the predetermined percentage of maximum power.
 4. The systemof claim 3, wherein the controller is adapted to total the valuesmeasured for each of a plurality of branch circuits and compare thetotal with a first overload value to detect an overload condition. 5.The system of claim 4, wherein the controller is adapted to send analarm signal to an operator upon detection of an overload condition. 6.The system of claim 5, wherein the controller is adapted to send asignal to disconnect power to one or more devices upon detection of anoverload condition.
 7. The system of claim 6, wherein the at least onecharacteristic is electrical current.
 8. The system of claim 7, whereinthe controller has a first network interface to communicate with devicespowered by the power distribution system over a first communicationsnetwork and a second network interface to communicate over a secondcommunications network; and wherein each of the plurality of powermonitoring devices includes a network interface to communicate with thecontroller over the second communications network.
 9. The system ofclaim 8, wherein the power distribution system includes anuninterruptible power supply, and wherein the controller is adapted tocommunicate with the uninterruptible power supply to detect that theuninterruptible power supply is operating on battery mode and replacethe first overload value with a second overload value.
 10. The system ofclaim 9, wherein the controller is adapted to send a signal to interruptpower to at least one device upon detection that the uninterruptiblepower supply is operating on battery mode.
 11. The system of claim 10,further comprising a plurality of temperature sensors that monitortemperature at locations within a facility, each of the temperaturesensors having an output to communicate a temperature value to thecontroller.
 12. The system of claim 11, wherein the controller isadapted to compare temperature values received from the temperaturesensors with predetermined values to detect an over temperature errorcondition, and upon detection of an over temperature error conditionsending an alarm signal.
 13. The system of claim 12, wherein thecontroller is adapted to send a signal to interrupt power to at leastone device upon detection of an over temperature error condition. 14.The system of claim 1, wherein the controller is adapted to compare thevalue measured with a predetermined value to detect an overloadcondition, and the controller is adapted to send a power interruptsignal to interrupt power to the device upon detection of an overloadcondition.
 15. The system of claim 14, wherein the controller is adaptedto control at least one device to operate at a reduced power level upondetection of an overload condition.
 16. The system of claim 14, whereinthe power monitoring device has a plurality of outlets, and the powermonitoring device is adapted to send a signal to the controller toindicate a value of the at least one characteristic for each of theplurality of outlets.
 17. The system of claim 16, wherein the powermonitoring device is adapted to control each of the plurality ofoutlets, and in response to the power interrupt signal from thecontroller, the power monitoring device is adapted to interrupt power toat least one of the plurality of outlets.
 18. The system of claim 1,wherein the at least one characteristic is electrical current.
 19. Thesystem of claim 1, wherein the controller has a first network interfaceto communicate with devices powered by the power distribution systemover a first communications network and a second network interface tocommunicate over a second communications network; and wherein each ofthe plurality of power monitoring devices includes a network interfaceto communicate with the controller over the second communicationsnetwork.
 20. The system of claim 1, wherein predetermined percentage ofmaximum power is 100 percent.
 21. The system of claim 7, wherein thepredetermined percentage of maximum power is 100 percent.
 22. A methodfor monitoring and controlling a power distribution system that has aplurality of circuit branches for providing power to a plurality ofdevices, the method comprising: controlling a first device on a firstcircuit branch to operate at a predetermined percentage of maximumpower; detecting a first value for a characteristic of power provided tothe first circuit branch using a first power monitoring device;controlling a second device on a second circuit branch to operate at apredetermined percentage of maximum power; detecting a second value fora characteristic of power provided to the second circuit branch using asecond power monitoring device; adding the first value to the secondvalue to obtain a total value; comparing the total value to an overloadvalue to detect an overload condition; indicating an alarm conditionwhen the total value exceeds the overload value.
 23. The method of claim22, wherein the first device is controlled to operate at less than thepredetermined percentage of maximum power when the second device iscontrolled to operate at the predetermined percentage of maximum power.24. The method of claim 23, further comprising controlling one of theplurality of devices to operate in a reduced power mode upon detectionof an overload condition.
 25. The method of claim 24, further comprisinginterrupting power to one of the plurality of devices upon detection ofan overload condition.
 26. The method of claim 25, wherein thecharacteristic measured is electrical current.
 27. The method of claim26, further comprising: communicating with the first device and thesecond device over a first communications network; and communicatingwith power detection devices over a second communications network. 28.The method of claim 26, wherein the power distribution system furtherincludes an uninterruptible power supply, and wherein the method furtherincludes steps of: detecting when the uninterruptible power supply isoperating in a battery mode; and controlling at least one of theplurality of devices to operate in a reduced power mode upon detectionof the battery mode.
 29. The method of claim 28, further comprisinginterrupting power to at least one of the plurality of devices upondetection of the battery mode.
 30. The method of claim 29, wherein thepower distribution system is at least partially contained within afacility, and wherein the method further includes steps of: measuringair temperature at a plurality of locations within the facility;comparing at least one value of air temperature measured with apredetermined value to detect an over temperature condition; andcontrolling at least one of the plurality of devices to operate in areduced power mode upon detection of the over temperature condition. 31.The method of claim 30, further comprising interrupting power to atleast one of the plurality of devices upon detection of the overtemperature condition.
 32. The method of claim 25, wherein the powerdistribution system is at least partially contained within a facility,and wherein the method further includes steps of: measuring airtemperature at a plurality of locations within the facility; comparingat least one value of air temperature measured with a predeterminedvalue to detect an over temperature condition; and controlling at leastone of the plurality of devices to operate in a reduced power mode upondetection of the over temperature condition.
 33. The method of claim 32,further comprising interrupting power to at least one of the pluralityof devices upon detection of the over temperature condition.
 34. Themethod of claim 22, wherein the first device and the second device arecontrolled to simultaneously operate at the predetermined percentage ofmaximum level, and the method further comprises steps of: measuring thecombined maximum power draw for the first circuit branch and the secondcircuit branch; and comparing the measured combined maximum draw withthe total overload value.
 35. The method of claim 22, wherein thecharacteristic measured is electrical current.
 36. The method of claim22, further comprising: communicating with the first device and thesecond device over a first communications network; and communicatingwith power detection devices over a second communications network. 37.The method of claim 22, wherein the predetermined percentage of maximumpower is 100 percent.
 38. The method of claim 26, wherein thepredetermined percentage of maximum power is 100 percent.
 39. A systemfor monitoring and controlling a power distribution system that has aplurality of circuit branches for providing power to a plurality ofdevices, the system including: means for controlling each of theplurality of devices to operate at a predetermined percentage of maximumpower; means for detecting a value of a characteristic of power providedto each of the plurality of circuit branches.
 40. The system of claim39, further comprising means for comparing a total value of thecharacteristic with a predefined value to detect an overload condition.41. The system of claim 40, further comprising means for interruptingpower to at least one of the plurality of devices when an overloadcondition is detected.
 42. The system of claim 41, wherein thecharacteristic is electrical current.
 43. The system of claim 42,wherein the power distribution system includes at least oneuninterruptible power supply, and the system further comprises means fordetecting that the uninterruptible power supply is in a battery mode ofoperation, and means for adjusting the predefined value when theuninterruptible power supply is in the battery mode of operation. 44.The system of claim 43, further comprising means for detecting airtemperature values in a facility containing the power distributionsystem.
 45. The system of claim 44, further comprising means forcomparing the detected air temperature values with predeterminedtemperature values, and means for interrupting power to at least one ofthe plurality of devices when the detected air temperature values exceedthe predetermined temperature values.
 46. The system of claim 39,wherein the power distribution system includes at least oneuninterruptible power supply, and the system further comprises means fordetecting that the uninterruptible power supply is in a battery mode ofoperation, and means for adjusting the predefined value when theuninterruptible power supply is in the battery mode of operation. 47.The system of claim 39, further comprising means for detecting airtemperature values in a facility containing the power distributionsystem.
 48. The system of claim 47, further comprising means forcomparing the detected air temperature values with predeterminedtemperature values, and means for interrupting power to at least one ofthe plurality of devices when the detected air temperature values exceedthe predetermined temperature values.
 49. The system of claim 39,wherein the characteristic is electrical current.
 50. The system ofclaim 39, wherein the predetermined percentage is 100 percent.
 51. Thesystem of claim 42, wherein the predetermined percentage is 100 percent.